Design a N D Regularization of Neural Networks : T H E O P T I M a L Use of a Validation Set

In this paper we derive novel algorithms for estimation of regularization parameters and for optimization of neural net architectures based on a validation set. Regularization parameters are estimated using an iterative gradient descent scheme. Architecture optimization is performed by approximative combinatorial search among the relevant subsets of an initial neural network architecture by employing a validation set based Optimal Brain Damage/Surgeon (OBD/OBS) or a mean field combinatorial optimization approach. Numerical results with linear models and feed-forward neural networks demonstrate t he viability of the methods. INTRODUCTION Neural networks are flexible tools for function approximation and by expanding the network any relevant target function can be approximated [6]. The associated risk of overfitting on noisy data is of major concern in neural network design [a]. The objective of architecture optimization is to minimize the generalization error. The literature suggest a variety of algebraic generalization error estimators e.g., FPE [l], FPER [9], GEN [7], GPE [13] and NIC 1141. These estimates, however, depend on a number of statistical assumptions which can be quite hard to justify. Hence, many neural net practitioners resort to empirical methods for design and validation, the simplest being to base the design on a single separate validation set. For further discussion on empirical generalization assessment see e.g., [lo]. In this contribution we derive schemes for proper selection of neural net architecture and for estimation of regularization parameters using a simple validation set approach. In fact, one may view estimation of the optimal regularization as an alternative 0-7803-3550-3/96$5.00@ 1996